Supplementary material for Dependent Normalized Random Measures

This is the supplementary material for the ICML 2013 paper Dependent Normalized Random Measures by the same authors. A. Notation & Preliminary We list some of the notation used in this paper in Table 1 for reminder. A.1. Definitions For completeness, we restate the definition of MNRM and TNRM. We are given a Poisson process on a product space R×Θ×R with intensity measure ν(dw,dθ,da) (we will use the notation νr(dw,dθ) = ∫ R̃r ν(dw,dθ,da)), denote the corresponding Poisson random measure as N (dw,dθ,da), the constructions are then defined as follow: Mixed Normalized Random Measures (MNRM) μ̃r(dθ) = ∫ R×R̃r wN (dw,dθ,da), r = 1, · · · ,#R μ̃t(dθ) = #R ∑ r=1 qrtμ̃r(dθ) t = 1, · · · , T μt(dθ) = 1 Zt μ̃t(dθ) , where Zt = μ̃t(Θ) (1) Proceedings of the 30 th International Conference on Machine Learning, Atlanta, Georgia, USA, 2013. JMLR: W&CP volume 28. Copyright 2013 by the author(s). Dependent Normalized Random Measures Table 1. List of notation. Notation Description I or #R #regions for R, indexed by r T #times for the observations Lt #observations in time t stl latent variable indexes which atom the l-observation in time t belongs to. gtl latent variable indexes which region the l-observation in time t belongs to. (wrk, θrk) points/atoms in the Poisson process in region R̃r. When used to construct an NRM, sometimes we also call wrk as the jumps and θrk as the atoms Kr #atoms with observation in the NRM in region R̃r Mr mass parameter for the NRM in region R̃r ~ Xt observations in the NRM in time t ntrk #observations in time t attached to the k-th jump of the NRM in region R̃r Nt total number of observations in time t n·rk = ∑ t ntrk ut auxiliary variable for the NRM in time t F (·|θrk) likelihood function under atom θrk N (w, θ) a Poisson random measure on W ×Θ ν(dw,dθ) Lévy measure for the NRM on R×Θ, we assume it is decomposed as ρ(dw)H(dθ), where H(·) is a probability measure on Θ. We use ν(dw,dθ,da) to denote the Lévy measure on the augmented space R × Θ × R and is assume to be factorized as ν′(dw,dθ)Q(da) Thinned Normalized Random Measures (TNRM) μ̃r(dθ) = ∫ R×R̃r wN (dw,dθ,da), r = 1, · · · ,#R zrtk ∼ Bernoulli(qrt), k = 1, 2, · · · μ̂t(dθ) = ∞ ∑ k=1 zrtkwrkδθrk , t = 1, · · · , T μt(dθ) = 1 Zt μ̂(dθ), where Zt = μ̃t(Θ) (2) A.2. Preliminary Lemmas We give three lemmas used in analyzing the properties and deriving the posteriors for the proposed MNRM, TNRM and their variants. Lemma 1 below is a celebrated formula for Lévy processes know as the Lévy-Khintchine formula. Lemma 1 (Lévy-Khintchine Formula) Given a completely random measure μ̃ (we consider the case where it only contains random atoms) constructed from a Poisson process on a produce space R × Θ with intensity measure ν(dw,dθ). For any measurable function f :W ×Θ −→ R, the following formula holds: E [ e−μ̃(f) ] M = E [ e− ∫ Θ f(w,θ)N (dw,dθ) ] = exp { − ∫ W×Θ ( 1− e−f(w,θ) ) ν(dw,dθ) } , (3) where the expectation is taken over the space of bounded finite measures. Using (3), the characteristic functional of μ̃ is given by φμ̃(u) M = E [ e ∫ Θ iuμ̃(dθ) ] = exp { − ∫ W×Θ ( 1− e ) ν(dw,dθ) } , (4) Dependent Normalized Random Measures where u ∈ R and i is the imaginary unit. Lemma 2 is about the disintegration property of a Poisson random measure N and some fixed points θk ∈ Θ. This is a specific result derived using either the Poisson process partition calculus (James, 2005), or the well known Palm formula. Lemma 2 Let N be a Poisson random measure defined on R × Θ with intensity measure ν(dw,dθ), μ̃ be the CRM constructed from N . Given samples {φn} with ties (θk)k=1 and the corresponding counts (n1, · · · , nK), for any nonnegative function f : R ×Θ 7→ R, the following formula holds: E [ e−N (f) K ∏ k=1 μ̃(θk) nk ] = E [ e−N (f) ] K ∏ k=1 ∫ R+ wk k e kkν(dwk, θk) , (5) where N (f) = ∫ R+×Θ f(w, θ)N (dw,dθ). Lemma 3, originally from Proposition 2.1 of (James, 2005), gives the posterior intensity measure of the Poisson process under an exponential tilting operation. It is used in the proof of the posterior Lévy measure for MNRM and TNRM. Lemma 3 Let N denotes a Poisson random measure with intensity measure ν, taking values in space of boundedly finite measures M with sigma-field denoted as B(M). BM+(W) denotes the collection of Borel measurable functions of bounded support on W. Then for each f ∈ BM+(W) and each g on (M,B(M)), ∫ M g(N )e−N P (dN|ν) = LN (f |N ) ∫ M g(N )P (dN|e−fν) , where P (dN|e−fν) is the law of a Poisson process with intensity e−f(w)ν(dw), LN (f |N ) = exp { − ∫ W ( 1− e−f(w) ) ν(dw) } denotes the Laplace functional of N . In other words, exponential tilting of a Poisson random measure as e−N P (dN|ν) is equivalent to dealing with a Poisson random measure with intensity e−fν. A.3. Normalized Generalized Gamma Processes In this subsection we briefly introduce a special class of normalized random measures called the normalized generalized Gamma process (NGG), and list some of its well known properties. A NGG is defined by normalizing a generalized Gamma process (GGP), whose Lévy measure ν(dw,dθ) is defined on the produce space R × Θ with the following form: ν(dw,dθ) = σM Γ(1− σ) w−σ−1e−wdwH(θ)dθ , (6) where 0 < σ < 1 is called the index parameter, M ∈ R is called the mass parameter, and H(·) is a probability measure on space Θ, called the base distribution. We will use NGG(σ,M,H(·)) to denote a NGG in the rest of the paper. We give the Laplace functional and the marginal posterior of the NGG below. These results can be used in the following sections. Lemma 4 (Laplace Functional of a GGP) For a generalized Gamma process μ̃g with Lévy measure defined in (6), let f : R 7−→ R be a measurable function, the Laplace functional of μ̃g is given by L(f |μ̃g) M = E [ e−μ̃g(f) ] = exp { − ∫ W×Θ ( 1− e−f(w) ) ν(dw,dθ) } f(w) M =uw −−−−−−→ exp {−M ((1 + u) − 1)} , where μ̃g(f) = ∫ R+×Θ f(w)N (dw,dθ), and u > 0 is a real constant. The Lévy measure of GGP can be formulated in different ways (Favaro & Teh, 2012), some via two parameters while some via three parameters, but they can be transformed to each other by using a change of variable formula. We only consider the form (6) in this paper for simplicity. Dependent Normalized Random Measures The following posterior result of a NGG is taken from (Corollary 2 Chen et al., 2012a), similar results can also be found in other references such as (James et al., 2009; Favaro & Teh, 2012). Lemma 5 (Posterior of a NGG) Let ~ X = (x1, · · · , xN ) be samples from the NGG(σ,M,H(·)) with distinct values (ties) (x1, · · · , xK) and the corresponding counts (n1, · · · , nK). Introduce a latent variable u (called latent relative mass (Chen et al., 2012a)), the marginal posterior is given by: p ( ~ X, u,K |σ,M ) = uN−1 Γ(N)(1 + u)N−Kσ (Mσ) K eM−M(1+u) σ K ∏ k=1 (1− σ)nk−1H(xk) , where (1− σ)nk−1 = (1− σ) · · · (nk − 1− σ) if nk > 1, and 1 if nk ≤ 1. B. Properties of MNRMs and TNRMs We have the following property for the MNRM. Proposition 6 (Proposition 1 in the main text) Conditioned on the weights qrt’s, each random probability measure μt defined in (1) is marginally distributed as a NRM with Lévy intensity ∑#R r=1 νr(w/qrt, θ)/qrt. Proof First, from the definiton we have μ̃t = #R ∑ r=1 qrtμ̃r . Because each μ̃r’s is a CRM, we have for any collection of disjoint subsets (A1, · · · , An) of Θ, the random variables μ̃r(An)’s are independent. Moreover, since the μ̃r’s are independent, we have that {μ̃t(Ai)}i=1 are independent. Thus μ̃t is a completely random measure. To work out its Lévy measure, we calculate the characteristic functional of each random measure qrtμ̃r using Lemma 1: φqrtμ̃r (u) = e − ∫ R+×Θ(1−e iuqw)νr(w,θ)dwdθ, = e− ∫ R+×Θ(1−e )νr(w/qrt,θ)dw/qrtdθ , where the last step follows by using a change of variable w′ = qrtw. Because qrtμ̃r’s are independent, we have that the characteristic functional of μ̃t is φμ̃t(u) = #R ∏ r=1 φqrtμ̃r (u) = e− ∫ R+×Θ(1−e ) ∑#R r=1 νr(w/qrt,θ)dw/qrtdθ , (7) The Lévy intensity of μ̃t is thus ∑#R r=1 νr(w/qrt, θ)/qrt. The following two properties are proved for TNRMs. Proposition 7 (Proposition 2 in the main text) Conditioned on the set of qrt’s, each random probability measure μt defined in (2) is marginally distributed as a normalized random measure with Lévy measure ∑ r qrtνr(dw,dθ). Proof One approach is to follow the proof of Lemma 11 in (Chen et al., 2012a), here we give a simplified proof using the characteristic function of a CRM (4). Denote B = {0, 1}#R×T , from the definition of μ̃t, the underlying point process can be considered as a MarkPoisson process in the product space R ×Θ×R×B, where each atom (w, θ) in region Rr is associated with a Dependent Normalized Random Measures Bernoulli variable z with parameter qrt. From the marking theorem of a Poisson process we conclude that μ̃t’s are again CRMs. To derive the Lévy measures, denote dz as the infinitesimal of a Bernoulli random variable z, using the Lévy-Khintchine formula for a CRM as in Lemma 1, the corresponding characteristic functional can be calculated as E [ e ∫ Θ iuμ̃t(dθ) ] = exp { − ∫ R+×Θ×R×B ( 1− e ) ν(dw,dθ,da)dz } = exp { − ∫ R+×Θ×R ( 1− e ) qratν(dw,dθ,da) } (8) = exp { − ∫ R+×Θ ( 1− e )(#R ∑ r=1 qrtνr(dw,dθ) )} , (9) where (8) follows by integrating out the Bernoulli random variable z with parameter qrat, (9) follows by integrating out the region space. Again according to the uniqueness property of the characteristic functional, μt’s are marginally normalized random measure with Lévy measures ∑#R r=1 qrtνr(dw,dθ). Proposition 8 (Proposition 3 in the main text) Denote the Lévy measure in region Rr as νr(dw,dθ), and fix the subsampling rates qrt. Given observations associated with a set of weights W , and auxiliary variables ut for each t ∈ T , the remaining weights in region Rr are independent of W , and are distributed as a CRM with Lévy measure ν′ r(dw,dθ) = ∏ t ( 1− qrt + qrtet ) νr(dw,dθ) . Proof The independence of the atoms with and without observations directly follows from the property of the completely random measures (James et al., 2009). It remains to proof the Lévy measure of the random measure formed by the random atoms of the corresponding Poisson process. The way to prove the posterior Lévy measure is to apply Lemma 3, where the idea is to formulate the joint distribution of the Poisson random measure and the observations into an exponential tilted Poisson random measure. Note it suffices to consider one region case because the CRM between regions are independent. For notational simplicity we omit the subscript r in all the statistics related to r, e.g., ntrw is simplified as ntw. Now denote the base random measure as μ̃, then construct a set of dependent NRMs μt’s by thinning μ̃ with different rates qj . Given observations for μt’s, by the Poisson partition calculus (James, 2005) it follows that the joint distribution for {μt} and observations with statistics {ntw} is p({ntw}, {μt}) = ∏ t ∏ k w ntwk k ( ∑ k′ ztk′wk′) Nt P (N|ν) . Now we introduce an auxiliary variable ut for each t via Gamma identity, and the joint becomes p({ntw}, {μt}, {ut}) = ∏ t ∏ k:ntwk>0 w ntwk k Γ(Nt) ∏ k e− ∑ t tktkP (N|ν) . Now integrate out all the ztk’s in the exponential terms we have: E{ztk} [∏